{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# textgenrnn 1.2 Avoiding Overfit\n",
    "\n",
    "by [Max Woolf](http://minimaxir.com)\n",
    "\n",
    "*Max's open-source projects are supported by his [Patreon](https://www.patreon.com/minimaxir). If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro\n",
    "\n",
    "You can use textgenrnn for text transfer learnin by training on one text dataset, and then another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from textgenrnn import textgenrnn\n",
    "\n",
    "textgen = textgenrnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2,000 texts collected.\n",
      "Training new model w/ 2-layer, 128-cell LSTMs\n",
      "Training on 114,560 character sequences.\n",
      "Epoch 1/5\n",
      "895/895 [==============================] - 125s 139ms/step - loss: 2.8554\n",
      "Epoch 2/5\n",
      "895/895 [==============================] - 125s 139ms/step - loss: 2.1527\n",
      "Epoch 3/5\n",
      "895/895 [==============================] - 123s 138ms/step - loss: 1.8864\n",
      "Epoch 4/5\n",
      "895/895 [==============================] - 122s 136ms/step - loss: 1.7128\n",
      "Epoch 5/5\n",
      "895/895 [==============================] - 125s 139ms/step - loss: 1.5834\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Trump to doggo soggo in the look for have a say a heckin comment to doggo\n",
      "\n",
      "Trump to doggo doggo does a heckin boop so so so compent to the work a big doggo\n",
      "\n",
      "S A L E B O Y E\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "S A N K B O Y E\n",
      "\n",
      "Donald Trump to get nee security claims compary to list to be a heckin bamboozle and there House of the White House on Trump Will Advestions Comey to Send Politice Assiguted Commentice Defing To Russia Real Alacause Have Advisers From Trump Passions Say Methice Office Will Resign Attroced Trump Se\n",
      "\n",
      "Trump New Secrets Segutive Who Sessions to Press the Berana Dies Feting President Trump To Lead Politics Post Trump\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Twert Ferrassendoy\n",
      "\n",
      "The seut Trump to record Trump Pare\n",
      "\n",
      "Wambards cot frendone make evering made some for Strighh+50 in White Hos\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_path = \"../datasets/reddit_rarepuppers_politics_2000.txt\"\n",
    "\n",
    "textgen.reset()\n",
    "textgen.train_from_file(file_path, new_model=True, num_epochs=5, gen_epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify a `train_size` to train on a subset of sequences, and prevent the neural network from learning *exact* sequences. The remaining data will be used as the validation set.\n",
    "\n",
    "Validation loss tests the efficiency of the model on sequences that the model has not seen. Ideally, validation loss should not *increase* after each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2,000 texts collected.\n",
      "Training new model w/ 2-layer, 128-cell LSTMs\n",
      "Training on 92,037 character sequences.\n",
      "Epoch 1/5\n",
      "719/719 [==============================] - 113s 157ms/step - loss: 3.1798 - val_loss: 2.6123\n",
      "Epoch 2/5\n",
      "719/719 [==============================] - 114s 158ms/step - loss: 2.3799 - val_loss: 2.2353\n",
      "Epoch 3/5\n",
      "719/719 [==============================] - 120s 167ms/step - loss: 2.0878 - val_loss: 2.0528\n",
      "Epoch 4/5\n",
      "719/719 [==============================] - 120s 167ms/step - loss: 1.9040 - val_loss: 1.9702\n",
      "Epoch 5/5\n",
      "719/719 [==============================] - 119s 166ms/step - loss: 1.7722 - val_loss: 1.9000\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Trump canss to the lies with a shour a heckin bamboozle\n",
      "\n",
      "Trump as a contern't to the sencit a heckin good the bamboozled the report\n",
      "\n",
      "S P O D B O Y E does a heckin good a heckin bamboozle\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "heckin up for sist sheet hill hoot fren to entraige of Trump's want a tolling and a potching stop the shurgey\n",
      "\n",
      "Donald Trump a amparacion from Russian probe and the mown change for from has talked and day be a heckin bamboozle\n",
      "\n",
      "Senate Donald Trump Is In State Dibation Interney Andering Attacking Intraved Prouction in Alemoust Counce Intested Donald Trump’s Wated No Source For His Moore Kecort\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Bussald: 5zar revise new 'with White Hoe Riby\n",
      "\n",
      "Trump: Clay's Wime Russia times news is redeates. Bexicre treadryma\n",
      "\n",
      "Donald Trump’s Russia evesticials into Trump. Pay lotly: '\"\"\"\"Reacun-offore\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_path = \"../datasets/reddit_rarepuppers_politics_2000.txt\"\n",
    "\n",
    "textgen.reset()\n",
    "textgen.train_from_file(file_path, new_model=True, num_epochs=5, gen_epochs=5, train_size=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, you can add a `dropout`, which drops out that proportion of characters in a sequence for a given epoch. This forces the model to weigh remaining characters more efficiently.\n",
    "\n",
    "Only use if `max_length` is not low, and don't set higher than `0.2` or the model may fail to converge!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2,000 texts collected.\n",
      "Training new model w/ 2-layer, 128-cell LSTMs\n",
      "Training on 91,516 character sequences.\n",
      "Epoch 1/5\n",
      "714/714 [==============================] - 113s 158ms/step - loss: 2.9999 - val_loss: 2.4777\n",
      "Epoch 2/5\n",
      "714/714 [==============================] - 115s 161ms/step - loss: 2.2966 - val_loss: 2.1371\n",
      "Epoch 3/5\n",
      "714/714 [==============================] - 115s 162ms/step - loss: 2.0338 - val_loss: 1.9792\n",
      "Epoch 4/5\n",
      "714/714 [==============================] - 112s 156ms/step - loss: 1.8684 - val_loss: 1.8907\n",
      "Epoch 5/5\n",
      "714/714 [==============================] - 111s 156ms/step - loss: 1.7467 - val_loss: 1.8504\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Trump says to lead to decure and the weet to the was to the president to report to report to dect to report to repeal to report to secret to concents to probe to the weet to report to pected a big to be a bill of the with the president in the president to prome the lead to the Russia intervettion \n",
      "\n",
      "Pupper does a heckin good corgo to seckin good boye\n",
      "\n",
      "Trump says to secret to the was to report to report confirms to repeal in the Trump compaign conting to repeal in the Is and the president in the Report\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Pupper boye does a sence pupper.\n",
      "\n",
      "Doggo hands loggo president cancered in the under says and will to seecking to sure the Browwer plessle poss\n",
      "\n",
      "Mect Specions of Trump Admition in Americans in the President Trump Trump in proped to comprotions it a sapper leader to record contricted Trump with Russia investigation in secret\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "dick flrumboye shout margers about and mashiria is lenrutic!\n",
      "\n",
      "This the Reporttt C C A O L A K B O Y E\n",
      "\n",
      "In's the porime vie: Dutcruad in indues ex-On I’m coun sorence lave wants forem tapting\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_path = \"../datasets/reddit_rarepuppers_politics_2000.txt\"\n",
    "\n",
    "textgen.reset()\n",
    "textgen.train_from_file(file_path, new_model=True, num_epochs=5, gen_epochs=5, train_size=0.8, dropout=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LICENSE\n",
    "\n",
    "MIT License\n",
    "\n",
    "Copyright (c) 2018 Max Woolf\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "of this software and associated documentation files (the \"Software\"), to deal\n",
    "in the Software without restriction, including without limitation the rights\n",
    "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "copies of the Software, and to permit persons to whom the Software is\n",
    "furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all\n",
    "copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
